Natural Language Parsing as Statistical Pattern Recognition

ii I certify that I have read this dissertation and that in my opinion it is fully adequate, in scope and in quality, a s a dissertation for the degree of Doctor of Philosophy. I certify that I have read this dissertation and that in my opinion it is fully adequate, in scope and in quality, a s a dissertation for the degree of Doctor of Philosophy. I certify that I have read this dissertation and that in my opinion it is fully adequate, in scope and in quality, a s a dissertation for the degree of Doctor of Philosophy. Preface The debate between the relative merits of statistical modeling and linguistic theory in natural language processing has been raging since the days of Zellig Harris and his irreverent student, Noam Chomsky. I h a v e never been shy about expressing my views on this issue. I would have liked nothing more than to declare in my dissertation that linguistics can be completely replaced by statistical analysis of corpora. In fact, I intended this to be my thesis: given a corpus parsed according to a consistent s c heme, a statistical model can be trained, without the aid of a linguistics expert, to annotate new sentences with that same scheme. The most important part of this thesis is that linguists need not participate in the development of the statistical parser. Using the most obvious representations of the annotations in the parsed corpus, the parser should automatically acquire disambiguation rules in the form of probability distributions on parsing decisions. In other words, natural language parsing would be transformed from the never-ending search for the perfect grammar into the simple task of annotating enough sentences to train rich statistical models. 1 With the guidance and support of the statistical modeling gurus in the IBM Speech Recognition Group, I formulated and implemented a statistical parser based on this thesis. In experiments, it parsed a large test set 1473 sentences with a signiicantly higher accuracy rate than a grammar-based parser developed by a highly-respected grammarian. The grammarian spent the better part of a decade perfecting his grammar to maximize its score on the crossing-brackets measure. 2 The grammarian's score 1 This task is not so simple if enough sentences" turns out to be, say, 10 trillion, but I will deal with that issue later. 2 For the deenition of the crossing-brackets measure, see …